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Congratulations 

• Your game 
launches on 
Tuesday! 

• But wait... 
something is 
wrong... 
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Launch Experiences 
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What are we Covering? 

• Runway to Launch 

• Materials 
Best Practices 

• Launch Window 

• Triage Flow 

Live Ops Dashboard 

• Questions in wrap up room 




Runway to Launch 

Preparing for take off 
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Angry Best Buy Guy 


Customer Support 

gathered more info: 

• PC preorder 
consumers at retail 

• Limited Edition copies 

• Missing bonus content 
when installing the 
game 


CAUTION 


ANGRY GAMER 


GHME 

OVER 
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Communication Chain 
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Define Your Partners 
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Recommended Materials 

• 24x7 Coverage plans 

• Countdown to Launch Calendar 

• Data Dictionary 

• Digital Offering and Unboxing Guide 

• End-to-End Flows 



GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


Other Best Practices 

Recommendations 

• War Room 

• Kill Switches 

• Localized Support 

• Onboard 3 rd Parties 






Launch Window 

Dealing with Live Issues 


I 
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What were our Challenges? 

• Unclear support and escalation paths 

• Little awareness of partner issues 

• No aggregate list of issues 

• No common format for reporting status 
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Our Solutions 

Unclear support and 

escalation paths 

• Define shifts for the title, 
usually for 2 weeks 

• Gather names and contact 
details with each partner 

• Make this data accessible 

• Train everyone on triage 
flow 
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Team 

Name 

Role 

Email 

Mobile 

Desk 

Location 

1 

Some Guy 

Online SE 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC 

1 

Some Guy 

Producer 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC 

1 

Some Guy 

DD 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC 

2 

Some Guy 

Online SE 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC 

2 

Some Guy 

Producer 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC 

2 

Some Guy 

DD 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC 

3 

Some Guy 

Online SE 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC -On Call 

3 

Some Guy 

Producer 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC -On Call 

3 

Some Guy 

DD 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC -On Call 


Some Guy 

SDD 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC - Escalation 1 


Some Guy 

EP 

someeuv@ea.com 

+1-999-999-9999 

+1-999-999-9999 

EAC - Escalation 2 



Special notes: Please always include "My Special Mailing List", special@ea.com, 

in any emails 
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Our Solutions 

Little awareness of 

partner issues 

• Put all partner team 
reps into the triage 
flow 

• Setup shift handoff 
calls to sync status 

• Run a war room 
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Our Solutions 

No aggregate list of 
issues or common 
reporting format 

• Accessible Confluence 
& JIRA 

• Reduces manual work 
for tracking and 
reporting 
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None 
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Triage Flow 


People Finding 
Issues 

QA 

SoCOM 
WWCE 
NOC 

You 


. Send an email 
\ using template/ 


Auto Creates JIRA Issue 


-H LRJIRA 


JIRA triggers email notification 


Enter issue 
into JIRA 


Automatic Reports 


LRDash&gardK- 
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Triage may require speculative issues 
to be researched by a team like QA to 
prove they are bugs before priority 
and ownership can be set 


Team triage 
tickets 


Optionally copy to 
other teams' systems > 
such as DevTrack 


Various 

internal 

issue 

trackers 


Update Status 



Leads track 
progress of fixes 


r 

LD/Shift Lead 
updates shift 
summary report 
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Live Ops Dashboard 


Report Live Ops Issue 


Update: 5-Mar-14 4PM PST 

Next Update: 6-Mar-14 6PM PST. Plants vs. Zombies - Garden Warfare Status Update Summaries 

Launch Director’s Status 

Teams: Game | Marketing 1 1 QA 1 1 Publishing 1 1 EADP || DPIO || Origin || Systems Test || WWCE 1 1 Community | Analytics | Commerce 1 1 Legal ] 

4 

Techs: | Blaze || Origin || Identity 1 1 Commerce | j Data Infrastructure | MS || Sony | 

Scale 0-5 

Features: Multiplayer Commerce Companion Apps Website DLC/Updates 


Notes 


• Analytics: Reporting on live data is improving. -IN PROGRESS 

• EADP and Analytics are addressing, with assistance from Game, GOS and DPIO. 

• Game server updates have been rolling out to fix some issues. 

• Xbox One co-op data has been missing, which has been tracked down to a client bug which means waiting until the next patch in April to resolve. 

• EADP/Blaze/Multiplayer: Players reported not being able to join or create game sessions on Xbox One and Xbox 360. -NO LONGER USER IMPACTING 

• GOS found an issue with some drivers for NIC's on some Blaze server hardware. This has been worked around overnight so we hope this resolves the problem. 

• Note that Garden Ops mode is peer-hosted, not using our dedicated game sewers, so users reporting issues with lag, dropping/getting kicked from games, etc. may be having issues with their connection to other players, which is to be expected in P2P 
games. Playing over a WiFi connection, for instance, is not recommended. 

• Game: Some players report problems with weapons and abilities unlocking. - IN PROGRESS 

• Game team have a script for fixing users in this state that will be run with each Blaze update/deploy. 

• Next Blaze deploy will include some changes that should make it less likely for users to get into this state. 

• Game: Some players are reporting that the Sticker Shop is unavailable or that they aren't receiving all of the coins they earn - IN PROGRESS 

• A handful of reports each day, but QA are unable to repro. 

• WWCE and Community teams have been asked to reach out to users to gather more specific info. 

• Game: Miscellaneous Gameplay Bugs - IN PROGRESS 

• Live QA have reported a small handful of gameplay bugs seen during retail playtesting. These are being tracked by the game team for fixing in future patches/DLC. A Microsoft VOIP issue preventing customers hearing each other in some circumstances is 
causing some comment. 
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Live Ops Dashboard 


• Live ua nave reporteo a small nanaiui or gameptay cogs $e«n aunng retail piaytesung inese are oemg iracKea oy ine game team nor nxmg in tuture patcneS'ULL a wrcroson vuik issue preventing customers neanng eacn omer in some circumstances is 
causing some comment 

A Notices 

• We re not releasing m Japan this week for either platform dually or packaged 

• The next server update deployment will happen at 2am PST on Friday 7-Mar 


Major Stats 

Q tf you require telemetry data or player stats please contact Colleen Chnsco who can support you in your efforts 
A Timezone for graphs is UTC 


Platform 

Xbox 3S0 


Xbox One 


24 Hour 


1 Week 



Blaze Metrics 

(Data up to 2-Mar-14. 23:59 GMT) 
LTD Master Accounts: 

? 

Daily Active Personas: 

? 

All Time PSU: 

? 

LTD Master Accounts: 

? 

Daily Active Personas: 

? 

All Time PSU: 

? 


0 > Click here for info on EADP central system usage 





GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


Live Ops Dashboard 

Critical Issue List 


PvZ Live Ops Open JIRA Issues List 





- 

Priority 

Summary 

Deliverable Status 

Comments/Resolution 

Functional Team 

Created 

Updated 

a 

[ XONE: Garden Warfare Issue - 'Unable to Join Session' 

On Track 

GOS deployed a Blaze fix at 1am PST on 1-Mar. Mur 

: EADP DevRel 

25/Feh/14 

05/Mar/14 

s 

Telemetry Issues - Large/Long telemetry files are arriving corrupted 

On Track 

Game, Data, GOS and DPIO are all looking at various 
New game server on 1-Mar seems to have fixed at lea 
Xbox One co-op data requires a client patch. 
Analytics are starting to get some more reporting bas 

Analytics 

20/Feb/14 

05/Mar/14 

a 

PvZGW XONE Abnormal Trend 

Not Started 


Launch Director 

08/Mar/14 

08/Mar/14 

a 

PvZGW XONE Abnormal Trend 

Not Started 


Launch Director 

08/Mar/14 

08/Mar/14 

a 

ERT Problem Alert - PvZ: Garden Warfare - Character Weapons Not Unlockina For Some Users 

Not Started 

DB script fixed up users with previous Blaze deploys. 

QA 

26/Feh/14 

06/Mar/14 


a 

APAC PVZ check Mar 11 

Not Started 


Launch Director 

1 1/Mar/14 

1 1/Mar/14 

a 

PVZGWXBL2 Unusual trend 

Not Started 


Launch Director 

1 1/Mar/14 

1 1/Mar/14 

a 

PyZ. GW . B laze, o ut age_f or. Thursday 3 AM PDT / 10 AM GMT 13th March 2014 

Not Started 


Launch Director 

1 1/Mar/14 

1 1/Mar/14 

a 

Sticker ShoD unavailable 

Not Started 

A handful of reports each day on answers.ea.com. Q> 
Game & QA have extra Q’s for WWCE/Community to 

i Game 

26/Feb/14 

05/Mar/14 


© Displaying 1 to 9 of 9 items 




Got an issue to report? This is how to do it, including a 24x7 Coverage Plan and 
Response Messaging 


Server usage graphs, telemetry reports. Blaze user metrics, WWCE Contact 
Reports and more 


Tracking events as they will happen around the world, such as website updates, game 
build propping, etc. 
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Results 

• Achieved our goals 

• Used by several teams 

• Others have had spin-off ideas 

• Another perspective 

• FIFA Live Team differences 



GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


Closing Thoughts 

• Key messages are simple: 

• Cross-functional planning 

• Open communication 

• Applies to any team, not just AAA or LR 


How does your team do it? 




Questions in Wrap Up Room 

Thanks for listening! 


Ashley Bennett, @ashleydb, abennett@ea.com 
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Get Ready for Launch 
How AAA Titles Do it 

Room 304, South Hall 
Today, 12:00pm - 12:30pm 

Ashley Bennett 

Technical Project Manager, EA 






